sequenceDiagram
  participant PythonWorker
  participant ChipWorker
  participant DeviceRunner
  participant Orchestrator
  PythonWorker->>ChipWorker: run_with_lease(slot_id, generation)
  ChipWorker->>DeviceRunner: select pipeline slot and arena bank
  DeviceRunner->>DeviceRunner: enqueue kernels and publish TASK_ACCEPTED
  DeviceRunner->>Orchestrator: mark task accepted
  Orchestrator-->>PythonWorker: acceptance wait completes

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

Rebased onto upstream/main@40b6329a and pushed fixes for the review findings. Head is now ee420e73 — one commit, 17 files, +773 −104.

Rebase: one commit was dropped, deliberately

git rebase dropped d3c60ba3 automatically (patch contents already upstream — it is #1539's content, which #1467's squash carried in). I then skipped 8023d914, and that one deserves an explicit note so it does not look like lost work.

8023d914 is the pre-review version of #1467. It still carries the transient state:

enum class MailboxState : int32_t {
    ...
    TASK_ACCEPTED = 8,
};

which loses the ACK whenever the child reaches TASK_DONE between two parent polls — exactly the short-task case the fence exists to pipeline. main (b072b5b9) has the fixed version: a sticky word at MAILBOX_OFF_ACCEPTED = MAILBOX_OFF_ERROR_MSG - 8, cleared only when the parent publishes the next TASK_READY. Keeping 8023d914 would have reverted that. Nothing unique to it is lost.

Findings addressed

1. Two independent generation authorities (was the blocker). PipelineSlotPool had no caller outside its own unit test; the mechanism that actually ran was ChipWorker::latest_pipeline_generations_, a monotone watermark that next_default_lease() advanced on every ordinary run. Since a fresh pool starts each slot at generation 1, any ChipWorker that had served ordinary runs rejected the pool's first lease. Verified on a2a3 before the fix:

>>> PROBE: slot 0 generation 1 REJECTED after 3 ordinary runs:
    run pipeline lease generation is stale

That is squarely on #1541's path — it acquires from pipeline_slots_.try_acquire(admission_depth_) in Orchestrator::begin_run().

Fixed by removing the second authority. next_default_lease() is gone; an unleased run takes slot 0 and advances nothing. ChipWorker now holds a PipelineSlotGenerationFilter whose contract is stated in the header: the pool mints and owns, the consumer only rejects replays of superseded generations. Regression: test_unleased_runs_do_not_consume_lease_generations.

2. The depth-two ST had no teeth. It asserted stream-set counts and generation rejection — both #1467/#1539 mechanisms already on main — and nothing about the resource split. Collapsing slot 1's Runtime buffer and arena bank onto slot 0 left all four tests green. Added ChipWorker::runtime_buffer_addrs and get_arena_bank_gm_heap_base_ctx so test_depth_two_slots_own_separate_resources can tell two copies from two names for one copy; the same mutant now fails it.

3. Slot/bank selection was thread-local. g_pipeline_slot / g_arena_bank were file-scope thread_local, so every DeviceRunnerBase on a thread shared one selection, finalize_common() did not reset it, and a per-runner array (retained_temp_addrs_) was indexed by a per-thread variable. Now pipeline_slot_ / arena_bank_ members, reset in finalize_common(). See the PR description for why this one has no end-to-end test.

4. runtime_bufs_ bypassed the contract this PR defines. It replicated by raw pipeline_depth while every other resource went through pipeline_resource_copy_count. tmr classifies RUNTIME_IMAGE as DEVICE_SCRATCH, so it was getting two host buffers for a resource its own contract says is shared — and at slot 1 would have read the second buffer while select_arena_bank(0) pointed the device at bank 0. Now derived from the classification.

5. _bank1_-suffixed members hardcoded depth 2 into identifiers. Six members plus seven bank == 0 ? x_ : x_bank1_ ternaries became std::array<std::unique_ptr<ArenaBank>, PTO_PIPELINE_MAX_DEPTH>. Raising the depth is now a constant change. retained_temp_addrs_ / retained_temp_sizes_ became std::array too (codestyle §8).

6. Bank 1 silently lost the prebuilt-arena cache. Both arena_bank_ != 0 early-returns now state that the cache holds one entry whose bases point into bank 0.

Not changed

select_pipeline_slot_ctx / select_arena_bank_ctx stay required (load_symbol, not load_optional_symbol). A stale build/lib/*.so then fails init() with a clear missing-symbol error instead of silently degrading, which matches the _task_interface build-stamp guard. The new get_arena_bank_gm_heap_base_ctx is optional, since it is pure diagnostics.

Full validation table is in the PR description.

Taking the correction: AICore streams are now created fresh for every Worker.run and retired on every exit path. Head is 6a6f7149.

You're right that the epoch was built on an unproven premise. It assumed that selecting a stream whose recorded image matches is as good as creating one; the only operation we have evidence for is creation. Tracking aicore_image_hash could then never make reuse safe, so nothing records it any more — the field stays for registration bookkeeping and carries no correctness duty.

• AICore stream: rtStreamCreate per run, rtStreamDestroy after reap through an RAII guard, so a failed launch or reap cannot leave one behind. ensure_run_stream_set refuses a slot that still holds one rather than reusing it.
• AICPU stream: unchanged, persistent per slot — it carries no instruction-cache state.
• run_stream_set_create_count now advances once per run; its docstring and docs/task-flow.md say so.

On the trace in your finding: it describes ee420e73. The intermediate ea9af7f4 did make its third step create a fresh stream — restoring the per-slot key made the new test fail with assert 7 == (7 + 1) — but that is moot under the per-run rule, and the epoch is gone.

Tests

The old assertions pinned the opposite invariant, so they inverted rather than extended:

test_depth_two_slot_is_generation_safe now also asserts a rejected lease creates no stream, since it must not reach launch.

Cost

This partly gives back #1464's persistent-AICore-stream gain, so here is what it measures at, on a 128×128 vector callable, 300 runs after 20 warm-ups:

Read that as an upper bound of roughly +1 ms/run, not a measurement: the two reuse builds differ by 0.78 ms and the fresh samples span 0.77 ms, so this shared box's run-to-run spread is the same size as the effect. The delta is one rtStreamCreate + rtStreamDestroy per run, so it is bounded and shrinks proportionally on larger workloads. A clean number needs an interleaved A/B, which needs both variants loadable in one process.

I think the trade is right — a guarantee that holds by construction beats a sub-noise host-time win — but flagging it explicitly because #1464 is merged and this reverses part of it. Say the word if you would rather keep same-image reuse and accept the narrower guarantee.

Also in this push

The three other findings from the previous round are unchanged and still in: host Runtime back to one buffer per slot (it is not the RUNTIME_IMAGE resource), has_serviceable_arena_topology rejecting unserviceable contracts at load, and sim refusing any slot but 0 instead of silently sharing its single arena set.

Validation on 6a6f7149: ctest 63/63, pytest tests/ut 843 passed / 2 skipped, run_stream_reuse 6/6 on a2a3. Sim and the full onboard sweep are running; I will post their results.

Head is 1b2ab0c2. All four findings addressed; PR description rewritten (it was still describing ee420e73).

[P1] sim declaring K=2 while rejecting slot 1 — took the first of your two options: sim now really implements K=2. SimDeviceRunnerBase got the same shape the onboard runner already has — one ArenaBank per slot in a std::array, per-slot retained_temp_addrs_, pipeline_slot_/arena_bank_ members — wired through sim's c_api_shared and released across all banks in both sim finalize() paths. My previous "sim refuses any slot but 0" was the wrong half of the either/or: it made the mismatch loud instead of removing it. The new TMR test leases slot 1 on sim and passes, which is the direct evidence.

[P2] generation filter is not an ownership check — correct, and your timing is exactly the hole: g1 released → g2 leased → g2 has not reached ChipWorker → delayed g1 still passes, because nothing raised the mark. Now stated where it can be found rather than only here — pipeline_slot_pool.h says it is "strictly weaker than an ownership check, and deliberately so", names the window, and points at gating dispatch on PipelineSlotPool::owns in the admission layer. docs/task-flow.md carries the same, replacing the old claim that "an older generation cannot reach launch". Also listed under Known boundaries in the PR description so B2 inherits it explicitly.

[P2] TMR slot-1 path uncovered — added tests/st/a2a3/tensormap_and_ringbuffer/pipeline_slots/. It interleaves slot0→slot1→slot0→slot1 leased runs, checking the result each time, that the two host Runtime buffers differ, and — TMR-specific — that slot 1 does not commit a second device-scratch bank, since TMR classifies its three pooled regions DEVICE_SCRATCH. 2/2 on a2a3, 2/2 on a2a3sim.

[P2] stale stream-reuse docs — swept: the contradictory docs/task-flow.md paragraph, the get_run_stream_set_create_count C ABI comment, the nanobind docstring, and the task_interface.py / worker.py properties. rg "reused only while|same-image runs reuse|is reused for every run" is now empty. PR description rewritten too — it had claimed TMR keeps one host Runtime buffer and that same-image runs reuse the stream, both opposite to the code.

On stream creation and prepare — agreed, no disagreement to record. rtStreamCreate is still inside DeviceRunner::run() ahead of launch_run, so it overlaps nothing. Moving it into a prepare stage, with the stream owned by {slot, generation} and destroyed on slot reuse / prepare failure / cancel plus ownership re-verified before launch, is async-preparation work. Listed under Known boundaries rather than implied to be done.

Two findings that were about main, not this diff — filed rather than resolved away: #1555 (_wait_for_acceptance assigns outside try/finally) and #1556 (producer-poisoned slots inflate pending_accepts). On #1556 I did adjust the severity: it does not strand the fence, since acceptance_ready() has an is_terminal disjunct — it delays the fence from acceptance to completion, which silently serialises the next submit().

On churn — 1092 lines noted. The split you suggest (contract/lease/filter vs platform slot resources + stream realization) is a clean seam; happy to do it before merge if you want it split.

Validation on 1b2ab0c2: ctest 63/63, pytest tests/ut 843 passed / 2 skipped, a2a3sim examples tests/st exit 0, a2a3 run_stream_reuse 6/6, a2a3 pipeline_slots 2/2. Full onboard sweep re-running against this head.

Head is af454c75. 三条全部落实。

1. rtStreamDestroy 失败回归测试

把 slot 状态机抽成 RunStreamSlots（src/common/platform/include/host/run_stream_slots.h），stream 的 create/destroy 用 std::function 注入 —— 是依赖注入，不是新的行为门控，所以不触发 env-macro-gating 那条规则。DeviceRunner 用 rtStreamCreate / rtStreamDestroy 构造它，生产行为不变。

tests/ut/cpp/hierarchical/test_run_stream_slots.cpp（no-hardware，6 个用例）逐条覆盖你要求的三点：

fake.fail_next_destroys(1);
EXPECT_EQ(slots.retire_aicore(0), -13);   // 1. 错误返回
EXPECT_EQ(slots.aicore(0), stranded);     // 2. 句柄保留
EXPECT_NE(slots.acquire(0), 0);           // 3. 下一轮拒绝复用
EXPECT_EQ(slots.destroy_all(), 0);        //    teardown 重试成功

变异验证：把 retire_aicore 回退成「失败也清句柄、吞错误」，第 99 / 102 / 105 行分别报错，正好一一对应这三项，而不是笼统挂掉。另外还覆盖了：被卡住的 slot 不拖累另一个 slot、destroy_all 只保留销毁失败的那条、create 失败不留半拥有状态、越界 slot 被拒。

2. 旧文档清理

rg "runner-scoped retained|single run stream set|reused only while|same-image runs reuse|Simulation serves slot 0" 现在为空。onboard runner header 的「同 image 复用」在上一轮已改。

3. PR 描述

Validation 表已改为 af454c75 的结果（ctest 64/64，含新目标），变异表加了这一轮的 rtStreamDestroy 项。

仍未结案：flat 形态下的段错误

CI 形态 sweep 跑到 L2 tensormap_and_ringbuffer 的 84%（零 FAIL、零段错误）后被 task-submit daemon 重启打断，不是超时也不是测试失败。剩下 16% 正在补跑，结果会单独贴。相关的 main 侧失败已由 #1561 解释（四个 notify demo 硬编码平台与设备 0/1），但那不解释分支侧的段错误，所以我没把它并进去。